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Abstract  - Real-world  data  fusion  systems  process  at  the 
lowest  level  and  combine  data  to  a usable  form  for  the 
human  at  the  highest  level.  The  functional  role  of  the  data 
fusion  system  is  to  provide  timely  and  accurate  data  to 
the  user.  The  focus  of  this  paper  is  to  address  the  user 
needs  for  a fusion  system  based  on  the  ontological 
principles  of  "becoming"  versus  the  historical  opposite  of 
"being”.  Ontology  is  a question  of  the  existence  of  the 
world,  but  also  equally  important  in  a philosophical 
argument  is  that  of  an  epistemology  - or  a set  of  rules. 
We  explore  the  higher-level  purpose  of  fusion  systems  by 
setting  up  the  world  and  emphasizing  rules  to  guide  a 
fusion  system  design  that  would  allow  for  human 
refinement  such  as  active  reasoning,  situation  awareness, 
and  pedigree  consistency. 

Keywords:  Fusion,  Ontology,  Epistemology,  Pedigree, 
Situational  Awareness 

1 Introduction 

The  Greeks  focused  on  both  data  fusion  and  the 
philosophical  questions  of  an  ontology  - the  study  if  the 
nature  of  existence.[  1]  For  data  fusion,  the  focus  was  on 
the  integration  of  senses  (i.e.  Aristotle)  [2],  The 
ontological  questions  focused  on  the  existence  of  man  and 
whether  or  not  man  could  leam  about  the  world  through 
sensing.  Thus,  the  combination  of  ontology  and  fusion 
were  united  in  the  aspects  of  how  the  human  interacted 
with  his  environment. 

One  of  the  major  issues  to  describe  the  existence  of 
humans  was  that  of  “being”  or  “becoming”.  Something 
that  is  unchanging  and  thus,  in  principle,  is  capable  of 
being  known  with  certainty  is  Being.  Being  implied  that 
we  exist  with  certainty,  and  that  man  exists  outside  the 
world.  Being  is  also  a philosophical  realization  that  man 
can  think  through  complex  issues  without  sensing  the 
world.  The  issues  surrounding  Becoming,  implied  the 
seeking  of  knowledge  and  truth  through  exploration  of  the 
world.  According  to  Heraclitus,  the  state  of  everything  in 
the  universe  is  dynamic  - that  is,  becoming  something 
other  than  what  it  was.[2]  One  of  main  groups  springing 
from  this  group  was  the  experimentalists  who  developed 
methods  of  statistics  which  is  fundamental  to  fusion 
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processing.  The  experimentalists  collected  data  and 
processed  it  to  understand  the  world  around  them.  Being 
implies  stability  and  certainty;  becoming  implies 
instability  and  uncertainty  in  a dynamically  changing 
world.  It  is  not  difficult  to  see  that  the  fusion  community 
subscribes  to  a “becoming”  philosophical  ontology. 

The  ontological  questions  surrounding  fusion  is  not  a 
new  question,  but  it  is  a fundamental  question  when  put  in 
the  framework  of  the  “purpose”  and  “functional  role”  of 
fusion  systems.  In  the  modem  day  exploration  of  the 
fusion  question,  the  thing  that  is  separates  the  Greek 
philosophical  questions  and  modem  day  fiisionists  is  the 
computer.  The  computer  can  process  large  amounts  of 
data  and  fusion  systems  seek  a reduction  is  dimensionality 
to  make  available  to  users,  information  that  can  be  acted 
on.  In  essence,  fusionists  must  address  an  ontological 
question  of  an  “extended”  sensing  fusion  ontology 
including  issues  associated  with  a human  interaction, 
information  pedigree,  and  situational  awareness. 

Some  of  the  fusion  questions  for  an  ontology  have  been 
explored  in  the  literature  and  an  example  is  from  the 
Fusion02  conference  there  are  common  fusion  questions 
be  addressed  by  an  ontological  reasoning  approach.  The 
DAML  (DARPA  Agent  Markup  Language)  program  is 
seeking  to  develop  a symbolic  ontology  for  situation 
awareness  [3],  From  the  ontological  questions  of  DAML, 
Kokar,  et  al  [3]  describes  an  ontology  for  recognition 
while  Capraro  et  al  [4]  address  a hardware  and  software 
ontology  to  deal  with  a large  amount  of  data  and 
taxonomy  to  assist  the  user.  When  dealing  with  a large 
database,  the  human  needs  the  ability  to  query  the 
information  needed  as  described  by  D’Agostino  in  a 
robotics  example  [5],  McDaniel  [6]  in  a command  and 
control  architecture,  Fransson,  et  al  [7]  in  a Situation 
Assessment  tool,  and  Wright,  et  al  [8]  in  an  inferencing 
engine.  To  explore  the  query  processing  by  the  human, 
Anken  [9]  and  Masters  [10]  explore  a schema  approach 
for  a question-answer  ontology  between  the  human  and 
the  fusion  systems.  The  papers  listed  above  show  a 
general  theme  of  the  fusion  community  to  explore  the 
higher  levels  of  information  fusion  which  includes  the 
human  as  a key  component  in  the  fusion  system.  Blasch 
and  Plano  [11]  have  described  the  interaction  between  the 
human  and  the  fusion  system  as  an  interaction  between  the 
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human  and  the  four  levels  of  the  JDL  fusion  model  with 
the  highest  level  (Level  5)  being  that  of  User  refinement. 
One  of  the  best  examples  of  user  issues  includes  situation 
awareness  by  Endsley  [12]  and  Roy  et  al  [13,  14]  which 
includes  elements  of  impact  assessment. 

This  paper  suggests  some  ontological  concerns  for 
data  fusion  with  a bias  towards  the  highest  level  of  data 
fusion  - user  refinement.  The  rest  of  the  paper  is  as 
follows:  Section  2 details  the  Greek  philosophical  ideals 
that  were  prevalent  in  the  first  ontological  debates  and 
outlines  for  the  reader,  the  purpose  for  the  ontological 
issues  surrounding  human-computer  interactive  fusion. 
Section  3 details  the  issues  higher  levels  of  information 
fusion.  Section  4,  utilizing  the  framework  as  the  human  as 
the  highest  level  of  information  fusion,  explores  the 
ontological  issues  surrounding  a fusion  system  including 
an  epistemology.  Section  5 summarizes  the  need  for  a 
focused  strategy  for  fusion  from  the  lowest  levels  of  data 
collection  to  information  processing. 

2 Early  Ontological  Questions 

In  his  search  for  truth,  Socrates  used  a method  called 
inductive  definition.  He  started  with  an  examination  of 
instances  of  such  concepts  as  beauty,  love,  justice,  or  truth 
which  are  hard  to  measure  and  quantify.  Socrates  then 
moved  on  to  questions  of  what  is  beauty  and  why  things 
are  through  a theory  of  essence.  The  essence  of  something 
is  its  basic  nature,  its  identifying,  enduring 
characteristics.^]  Using  the  argument  of  Socrates, 
fusionists  are  concerned  with  the  underlying  primitives  of 
concepts  we  seek  to  separate  for  decision  making. 

2.1  Defining  a taxonomy  of  knowledge 

Socrates  student,  Plato  postulated  a theory  of  forms  - 
the  pure,  abstract  realities  that  are  unchanging  and 
timeless  and  therefore  knowable.[15]  Such  forms  create 
imperfect  manifestations  of  themselves  when  they  interact 
with  material.  It  is  these  imperfect  manifestations  of  the 
forms  that  are  the  objects  of  our  sense  impressions.  Plato's 
contention  is  that  ultimate  reality  consists  of  abstract  ideas 
or  forms  that  correspond  to  all  objects  in  the  empirical 
world.  Knowledge  of  these  abstractions  is  innate  and  can 
be  attained  only  through  introspection. 

What  becomes  of  those  who  attempt  to  gain  knowledge 
by  examining  the  empirical  world  via  sensory  experience? 
According  to  Plato,  they  are  doomed  to  ignorance  or,  at 
best,  opinion.  The  only  true  knowledge  involves  grasping 
the  forms  themselves,  and  this  can  be  done  only  by 
rational  thought.  Plato  summarized  this  viewpoint  with  his 
famous  analogy  of  the  divided  line,  which  is  illustrated  in 
Figure  1.  Imagining  is  seen  as  the  lowest  form  of 
understanding  because  it  is  based  on  images  - for  example, 
a portrait  of  a person  is  once  removed  from  the  person. 
Intelligence  and  knowledge  are  the  highest  levels  of 
information  abstraction. 
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Figure  1:  Plato's  analogy  of  the  divided  line.  (From 
Comford's  translation  of  Plato's  Republic.  1968,  p.222.) 

Both  Plato  and  Aristotle  were  primarily  interested  in 
essences  or  truths  that  went  beyond  the  mere  appearance 
of  things,  but  their  methods  for  discovering  those  essences 
were  distinctly  different.  For  Plato,  essences  corresponded 
to  the  forms  that  existed  independently  of  nature  and  that 
could  only  be  arrived  at  by  ignoring  sensory  experience 
and  turning  one's  thoughts  inward  (that  is,  by 
introspection).  For  Aristotle,  essences  existed  but  could 
become  known  only  by  studying  nature.  He  believed  that 
if  enough  individual  manifestations  of  a principle  or 
phenomenon  were  investigated,  eventually  one  could  infer 
the  essence  that  they  exemplified.  For  Plato,  first 
principles  were  arrived  at  by  pure  thought;  for  Aristotle, 
they  were  attained  by  examining  nature  directly.  [2] 

2.2  Understanding  and  Reasoning 

Aristotle  believed  sensory  experience  to  be  the  basis  of 
all  knowledge,  although  the  five  senses  and  the  common 
sense  provided  only  the  information  from  which 
knowledge  could  be  derived.  Aristotle  also  believed  that 
everything  in  nature  had  within  it  an  entelechy  (purpose) 
that  determined  its  potential. [16]  Active  reason,  which 
was  considered  the  immortal  part  of  the  human  soul, 
provided  humans  with  their  greatest  potential,  and 
therefore  fully  actualized  humans  engage  in  active  reason. 
Because  everything  was  thought  to  have  a cause,  Aristotle 
postulated  an  unmoved  mover  that  caused  everything  in 
the  world  but  was  not  itself  caused. 

To  truly  understand  anything,  according  to  Aristotle,  we 
must  know  four  things  about  it.  That  is,  everything  has  the 
following  four  causes: 

1.  Material  cause  is  the  kind  of  matter  of  which  an 

object  is  made.  For  example,  a statue  is  made  of 
marble. 

2.  Formal  cause  is  the  particular  form  or  pattern  of 

an  object.  For  example,  a piece  of  marble  takes 
on  the  form  of  Venus. 

3.  Efficient  cause  is  the  force  that  transforms  the 

matter  into  a certain  form  - for  example,  the 
energy  of  the  sculptor. 

4.  Final  cause  is  the  purpose  for  which  an  object 

exists.  In  the  case  of  a statue,  the  purpose  may  be 
to  bring  pleasure  to  those  who  view  it.  The  final 
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cause  is  "that  for  the  sake  of  which  something 
exists."  A final  cause  (a  thing's  purpose)  actually 
precedes  the  other  three  causes. 

Using  the  ontological  debate  of  the  Greeks,  we  see  that 
fusionists  would  follow  Aristotle  when  defining  a purpose 
for  the  fusion  system  while  understanding  the  various 
causes  of  information. 

For  Aristotle,  as  for  most  Greek  philosophers,  a soul 
was  that  which  gives  life;  therefore,  all  living  things 
possess  a soul.  According  to  Aristotle,  there  are  three 
types  of  souls,  and  a living  thing's  potential  (purpose)  is 
determined  by  what  type  of  a soul  it  possesses. 

1.  A vegetative  (or  nutritive)  soul  is  possessed  by 

plants.  It  allows  only  growth,  the  assimilation  of 
food,  and  reproduction. 

2.  A sensitive  soul  is  possessed  by  animals  but  not 

plants.  In  addition  to  the  above  functions, 
organisms  that  possess  a sensitive  soul  sense  and 
respond  to  the  environment,  experience  pleasure 
and  pain,  and  have  a memory. 

3.  A rational  soul  is  possessed  only  by  humans.  It 

provides  all  the  functions  of  the  other  two  souls 
but  also  allows  thinking  or  rational  thought. 

For  a fusionist,  we  can  utilize  the  tripartite  soul 
approach  to  human  interaction  with  three  levels  of 
automatic,  associative,  and  cognitive  functions. 

2.3  Common  Sense  and  Active  Reason 

As  important  as  sensory  information  was  to  Aristotle,  it 
was  only  the  first  step  in  acquiring  knowledge.  In  other 
words,  sensory  experience  was  a necessary  but  not  a 
sufficient  element  in  the  attainment  of  knowledge.  In  the 
first  place,  each  sensory  system  provides  isolated 
information  about  the  environment  that  by  itself  is  not 
very  useful.  For  example,  seeing  an  object  move  provides 
a clue  as  to  its  condition,  hearing  it  move  provides  another 
clue,  smelling  it  may  give  a clue  as  to  why  it  is  running, 
and  touching  may  reveal  that  it  a decoy.  It  is  the  combined 
information  from  all  the  senses  that  allows  for  the  most 
effective  interactions  with  the  environment. 

Aristotle  postulated  a common  sense  as  the  mechanism 
that  coordinated  the  information  from  all  the  senses. [2] 
The  common  sense,  like  all  other  mental  functions,  was 
assumed  to  be  located  in  the  heart.  The  job  of  common 
sense  was  to  synthesize  sensory  experience,  thereby 
making  it  more  meaningful.  However,  sensory 
information,  even  after  it  was  synthesized  by  common 
sense,  could  provide  information  only  about  particular 
instances  of  things.  Passive  reason  involved  the  utilization 
of  synthesized  experience  for  getting  along  effectively  in 
everyday  life,  but  it  did  not  result  in  an  understanding  of 
essences,  or  first  principles.  The  abstraction  of  first 
principles  from  one's  many  experiences  could  be 


accomplished  only  by  active  reason,  which  was 
considered  the  highest  form  of  thinking.  Aristotle 
therefore  delineated  levels  of  knowing  or  understanding 
much  like  Plato's  divided  line: 

• Active  reason:  The  abstraction  of  principles,  or 

essences,  from  synthesized  experience 

• Passive  reason:  Utilization  of  synthesized 

experience 

• Common  sense:  Synthesized  experience 

• Sensory  information:  Isolated  experiences 

An  example  of  how  these  levels  of  understanding  are 
related  might  be  to  experience  a car  through  the  senses  of 
sight  (seeing  an  car  move),  vibration  (riding  in  a car),  and 
hearing  (hearing  the  engine).  These  experiences  would 
correspond  to  the  level  of  sense  reception.  The  common 
sense  would  indicate  that  all  these  experiences  had  a 
common  source  - car.  Passive  reason  would  indicate  how 
a car  could  be  used  in  a variety  of  practical  ways,  whereas 
active  reason  would  seek  the  laws  governing  a car’s 
motion  and  an  understanding  of  its  essence.  What  started 
as  a set  of  empirical  experiences  ends  as  a search  for  the 
principles  that  can  explain  those  experiences. 

The  active  reason  part  of  the  soul  provides  humans  with 
their  highest  purpose.  Just  as  the  ultimate  goal  of  an  seed 
is  to  become  an  tree,  the  ultimate  goal  of  humans  is  to 
engage  in  active  reason. 

3 Human-Fusion  Interaction 

One  fusion  model  that  has  been  utilized  by  fusion 
community  is  the  JDL  fusion  model,  which  has  been 
adapted  to  highlight  the  need  for  human  refinement  [11]. 
In  the  JDL-User  model,  the  focus  is  the  human’s  ability  to 
perform  active  reasoning  capabilities  over  the  other  fusion 
stages  to  seek  knowledge  about  the  world. 


Figure  1.  JDL-User  Model 
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In  the  model,  level  5 - User  refinement  is  described  as 
an  element  of  Knowledge  Management:  adaptive 

determination  of  who  queries  information  and  who  has 
access  to  information,  (e.g.  information  operations)  and 
adaptive  data  retrieved  and  displayed  to  support  cognitive 
decision  making  and  actions  (e.g.  altering  the  sensor 
display).  While  many  fusion  models  explore  the 
capabilities  of  the  human,  it  is  not  a question  of  where  the 
boxes  go,  but  what  is  the  focus  of  fusion  system. 

From  the  analysis  of  the  Greek  ontological  question, 
fusionists  subscribe  to  the  theory  of  a changing  world  with 
dynamic  objects  (Level  1).  A computer  is  good  at 
processing  large  amounts  of  object  data  while  a human  is 
good  at  inferencing  over  the  data.  Thus,  the  key  to  the 
human  refinement  question  is  the  “what  framework  the 
human  has  to  inference  over  on  the  data  and  what 
capabilities  allow  for  active  reasoning?” 

To  develop  the  user  requirements  or  refinements, 
three  issues  are  paramount  (1)  situational  awareness,  (2) 
prioritization  of  needs,  and  (3)  pedigree  information. 

3.1  Situational  Awareness 

The  Human  in  the  Loop  (HIL)  of  a semi-automated 
system  must  be  given  adequate  situation  awareness  (SA). 
According  to  a pioneer  and  continued  leader  in  the  SA 
literature,  Endsley  stated  that  "Situation  awareness  is  the 
perception  of  the  elements  in  the  environment  within  a 
volume  of  time  and  space,  the  comprehension  of  their 
meaning,  and  the  projection  of  their  status  in  the  near 
future.”  [12].  This  now-classic  model  translates  into  3 
levels: 

• Level  1 SA  - Perception  of  environmental  elements 

• Level  2 SA  - Comprehension  of  current  situation 

• Level  3 SA  - Projection  of  future  states 


Situation  Awareness  Model 

Ewfcky,  im 


Figure  2.  Endsley’s  Situation  Awareness  Model. 

Operators  of  dynamic  systems  use  their  situation 
awareness  in  determining  their  actions.  To  optimize 
decision  making,  the  SA  provided  by  a system  should  be 
as  precise  as  possible  as  to  the  objects  in  the  environment 


(Level  1 SA).  A situation  awareness  approach  should 
present  a fused  representation  of  the  data  (Level  2 SA)  and 
provide  support  for  the  operator's  projection  needs  (Level 
3 SA)  in  order  to  facilitate  operator's  goals.  From  the  SA 
model  presented  in  Figure  2,  workload  is  a key  component 
of  the  model  that  affects  not  only  SA,  but  also  the  decision 
and  the  action  of  the  user  in  performing  active  reasoning. 

Issues  associated  with  the  awareness  of  the  human 
are  (1)  workload,  (2)  attention  and  (3)  trust  [11],  In  the 
case  of  workload,  we  minimize  the  amount  of  information 
the  human  must  process  which  also  helps  to  highlight  the 
need  for  different  types  of  “cause”  information  for  the 
user.  Trust  will  be  discussed  in  the  efficacy  section; 
however  the  ontological  question  is  the  ability  to  convey 
information  on  the  computer  as  extended  sensory 
information,  about  the  truth  of  the  world.  False  data 
would  hinder  the  direct  connection  between  the  user  and 
the  world  around  them. 

3.2  Prioritization  of  Needs 

To  reduce  the  set  of  information  to  a dimensionally 
attractive  set  of  information  requires  that  a hierarchy  of 
needs  is  determined  by  the  human  processed  by  the  fusion 
system.  A hierarchy  of  needs  helps  to  allow  the  human 
to  perform  active  reasoning  over  the  data.  In  this  case, 
we  can  postulate  the  needs  by  addressing  the  impact 
assessment  (Level  3)  questions.  Some  of  the  needs  that 
can  be  included  in  a hierarchy  include: 

Things: 

Objects  - Number  and  types 
Threat  - whether  harmful,  passive,  or  helpful 
Location  - close  or  far 
Basic  primitives  - features 
Existence  known  or  unknown  - (i.e.  new  objects) 
Moving  or  static 
Processes: 

Measurement  system  reliability  (uncertainty) 

Ability  to  collect  more  data 
Delays  in  the  measurement  process 

It  is  the  user  that  must  interact  with  a fusion  system 
to  determine  the  priority.  The  priority  of  information  is 
related  to  the  information  sought,  but  to  provide  the 
human  with  the  ontological  ability  to  understand  the 
world,  they  must  have  the  ability  to  choose  or  select  the 
objects  of  interest  and  the  processes  from  which  the  raw 
data  is  converted  to  the  fused  data.  One  of  the  issues  in 
the  processing  of  fused  information  is  related  to  ability  to 
understand  the  origin  of  the  information  or  pedigree  of  the 
data  and  information. 

3.3  Pedigree  Information 

Pedigree  information  is  the  origin  or  history  of 
something  [1].  In  the  case  for  a fusion  system,  it  is 
important  to  understand  where  the  data  originated  from, 
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however,  the  availability  of  pedigree  information  is  not 
necessary  for  the  human  unless  there  is  a conflict  of 
information.  When  a conflict  arises,  the  human  needs  to 
be  able  to  drill  down  from  the  fusion  result  to  the  data 
processed.  This  not  only  includes  the  information  type 
and  collection  but  also  the  time  of  the  information. 
Understanding  when  the  data  was  transformed  into 
information  is  important  for  the  human  to  make  decisions 
on. 

Pedigree  information  ensures  that  the  data  being 
processed  is  consistent  and  traceable  to  ensure  the 
integrity  of  the  information.  Having  an  ontology  of 
information  leads  to  an  interoperability  between  systems 
since  the  consistency  of  one  system  is  related  to  the  next 
(see  the  section  on  taxonomy,  semantics,  and  efficacy). 
Some  of  the  issues  in  pedigree  include: 

Integrity  - formal  correctness  (i.e.  uncertainty  analysis) 
Consistency  - reducing  ambiguity  or  double  counting 
(i.e.  measurement  origination) 

Traceability  - backward  inference  chain  ( i.e.  inferences 
assessed  as  data  is  translated  into  information). 

In  the  case  in  which  pedigree  information  is  (1) 
decomposed  from  the  user  to  the  machine  and  (2) 
composed  from  the  machine  to  the  user,  below  are  listed  a 
pedigree  taxonomy  [17]  to  accomplish  a fusion  task: 

Decomposition  Pedigrees 

(A)  Information  Pedigree:  Links  information  needs 
decomposition  elements-of-information  from 
collection  task  to  cognitive  information  need 

Defines  System  Objective  Model  (SOM) 

flD  Context  Pedigree: 

Links  decomposition  decision  context  to  elements-of- 
information  in  SOM  (e.g.  situation,  predictive 
assumptions,  target  models,  doctrine) 

Composition  Pedigrees 

Source  Pedigree: 

Lineage  of  evidence  used  to  develop  fusion  products. 

“Drill  Down”  to  observation  source  in  Situation 
Estimate 

Processing  Pedigree: 

Links  context  used  to  “fuse”  data  with  uncertainty/error 
associated  with  the  hypotheses/observations  in  the 
inference  chain  of  the  source  pedigree. 

3.4  Utility  of  Pedigree  information 

Using  the  taxonomy  of  pedigree  information 
provided  above,  there  are  some  elements  of  the  utility  of 
the  information  that  should  be  addressed. 

Context  Pedigree  is  a decomposition  product  that 
is  essential  for  useful  exploitation  of  fusion  systems.  The 
context  can  be  a form  of  the  situation  assessment.  Having 
the  essential  context  information  is  required  for  consistent 


operational  assessment  of  dynamic  and  uncertain  data, 
which  was  mentioned  in  the  ontological  issues  associated 
with  the  existence  of  information.  Additionally,  context 
pedigree  requires  common  knowledge  of  sensor  and  data 
models  between  fusion  and  needs  decomposition  and 
requires  consistent  resource  performance  and  asset 
behavior  models.  While  having  the  correct  context 
information  is  needed,  having  the  correct  information 
within  the  context  framework  is  also  needed. 

Information  Pedigree,  which  can  be  derived  from  an 
Information  Needs  decomposition,  is  really  the 
decomposition  of  the  user  needs  into  actionable  and 
control  data  collection  for  fusion  results.  Having  the 
correct  information  enables  effective  assessment  of  new 
information  needs  and  focuses  fusion  processes  on  the 
goals  of  the  user.  The  information  pedigree  links  the  data 
collected  to  the  situation  assessment  and  the  data  base  of 
information  from  which  both  the  user  and  the  fusion 
system  reason  over.  Once  the  context  establishes  the 
information,  the  data  fusion  system  needs  a processing 
pedigree. 

Processing  Pedigree  is  a function  of  the  composition 
product  from  the  data  fusion  system.  The  processing 
pedigree  is  essential  for  anticipatory  resource 
management.  In  the  case  in  which  the  user  can 
proactively  affect  the  fusion  system,  the  user  needs  to  be 
able  to  interact  with  the  system  at  various  levels  of 
abstraction.  To  estimate  a marginal  fusion  gain  from 
differing  sensors,  geometry,  etc.,  the  user  must  have  the 
ability  to  know  at  what  stage  the  fusion  process  is  in. 
Processing  pedigree  requires  consistent  models  of 
phenomenology,  platforms,  and  sensors  as  well  as  requires 
traceability  of  derived  products  to  the  source. 

Source  Pedigree,  which  helps  in  the  analysis  of  the 
derived  results  from  information  composition  (i.e.  fusion), 
needs  to  be  maintained  in  situation  database.  Source 
pedigree  enables  understanding  of  degree  to  which 
information  satisfies  commander’s  needs  and  helps  the 
user  ‘Know  what  it  doesn’t  know’  to  drive  new 
information  needs.  Once  the  user  is  aware  of  the  source 
limitations,  they  can  be  linked  to  collection  database  so  as 
to  determine  if  other  data  exists. 

In  order  to  process  the  pedigree  information  from 
either  decomposition  or  composition,  metrics  need  to  be 
established  to  coordinate  the  validity  of  the  information. 
Fusion  processes  simultaneously  contribute  to  uncertainty 
metrics  associated  with  processing  and  context  pedigrees 
for  information  products  which  form  a basis  for 
uncertainty  calculus  to  predict  satisfaction.  Additionally, 
Processing  & Context  Pedigrees  need  to  be  semantically 
consistent  and  source  and  information  pedigree  require  a 
taxonomy  of  consistency  with  database  and  the  user. 

Some  additional  pedigree  topics  include:  Semantic 
pedigree,  age  pedigree,  and  categorical/taxonomy  pedigree 
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3.5  Human  refinement 

To  actively  reason  over  a complex  set  of  data,  a usable 
set  of  information  must  be  available  from  which  the 
human  can  act  on.  The  ontology  - the  seeking  of 
knowledge  and  the  determination  of  the  world,  [2] 
indicates  that  the  world  must  be  defined  and  the  next 
section  explores  issues  related  to  an  ontology  for  human 
refinement. 

4 Ontological  Issues 

In  seeking  the  knowledge  of  the  world  through 
measurement,  we  have  to  define  context  of  the 
information  we  are  interested  about.  To  create  an  analogy, 
we  use  a game  to  define  the  world  of  information  and 
processes  of  interest  for  which  the  user  inferences  over. 

4.1  Ontology 

Ontology  is  based  on  the  goal  of  the  system.  Utilizing 
the  same  arguments  from  the  Greeks  implies  that  we 
should  consider  the  purpose  or  existence  of  something.  In 
this  case,  we  can  divide  the  knowledge  problem  into  three 
areas  (1)  world  or  frame  of  reference,  (2)  fusion  processes, 
and  (3)  types  of  reasoning  ontology.  This  paper  explores 
only  one  idea  for  an  ontological  framework  and  there  are 
others  that  exist  until  falsified.  For  an  ontology  of 
knowledge,  it  is  noted  from  Plato’s  definitions  that 

Beliefs  are  many,  but  Knowledge  is  rare  and  focused  [20] 

In  designing  a sensor  fusion  system,  we  are  interested  in 
what  we  are  trying  to  accomplish,  which  can  be  framed  as 
a knowledge  game,  or  the  types  of  knowledge  we  are 
trying  to  seek.  An  analogy  is  that  if  we  are  playing  a 
game,  (1)  which  ballpark  are  we  playing?  For  a fusion 
system,  the  game  is  to  determine  the  objects  in  the  world  - 
their  location,  movement,  and  status.  Once  we  have 
decided  which  game  we  are  playing,  we  have  established 
the  ontology  of  the  fusion  system.  The  ontology  is  the 
declaration  of  world  and  its  constructs  (i.e.  natural  world, 
sensing).  Next  we  need  the  epistemology  (set  of  rules  to 
know  something)  of  the  game.  For  a game  analogy,  we 
have  to  know  whether  or  not  the  rules  we  are  playing  by 
would  (2)  allow  us  to  win  the  game?  To  win,  we  have  to 
define  the  fusion  rules  to  declare  that  we  have  won  - such 
as  determining  the  object  identity  within  some  probability 
of  quality. 

For  example,  if  Plato  wanted  to  know  something  to  win 
the  knowledge  game  he  postulated  three  things 

A)  Proposition  - has  to  be  true,  not  false  and  not  beliefs 

B)  Believe  in  the  truth  - agree  with  the  assumptions 

C)  Evidence  to  support  the  truth  (very  difficult  to  do) 

In  a fusion  system,  we  are  concerned  with  determining 
whether  or  not  we  have  accumulated  evidence  to  support  a 


proposition  in  something.  For  example,  in  a fusion 
system,  we  seek  features  to  determine  the  object  identity. 

If  the  information  gathered  does  not  combine  to  create  a 
confident  belief  in  the  proposition  of  an  object  as  a 
specific  target,  then  we  seek  additional  evidence  from 
other  sensory  data  to  improve  the  belief  in  a target  type. 
The  ontology  for  the  fusion  system  is  based  on  algorithms 
to  combine  data  and  thus,  we  move  from  the  knowledge 
game  to  the  science  game. 

The  second  game  that  we  play  is  the  science  game 
which  is  the  mathematics  and  the  data  collection  to 
determine  the  believability  of  the  fused  results.  We  might 
infer  that  we  have  an  ontology  of  a knowledge  game,  but 
rather  we  are  playing  a specific  science  game,  which  can 
be  referred  to  as  the  FUSION  GAME. 

The  fusion  game  has  a similar  structure  to  the 
ontological  questions  of  the  past.  In  the  case  of  a fusion 
game,  the  ontology  is  the  materialism  of  finding  objects 
and  feature  data  of  interest.  The  Epistemology  - rules, 
methods,  and  processes,  include  the  mathematical 
principles  of  the  game.  One  advantage  of  the  fusion  game 
is  that  nature  is  arbiter  of  the  game.  Nature  is  the  truth 
data  that  tells  the  fusionist  whether  or  not  the 
determination  of  information  is  correct.  Thus,  to  win  the 
fusion  game,  we  desire  no  anomalies  (i.e.  scientific 
knowledge  is  reproducible/  mathematical).  Thus,  if  the 
mathematics  is  developed  in  one  situation,  it  should  be 
portable  to  another  situation.  The  mathematical  principles 
cover  the  inductive  reasoning  over  information,  while  a 
deductive  empirical  analysis  comes  from  the  observations 
of  data  and  fused  results. 

Inductive  reasoning  is  the  determination  of  Plausible 
belief  - expectations  / predictions  - from  a set  of 
observations.  To  accomplish  the  inductive  reasoning,  a 
fusion  system  needs  to  gather  the  hypothesis  from  the  user 
and  data  to  test  the  hypothesis.  The  information  needs  to 
be  in  a framework  of  a testable  hypothesis  that  can  be 
verifiable  or  falsifiable  hypothesis.  To  be  able  to  verify  a 
hypothesis,  features  must  be  predictable  (i.e.  from  a 
model)  and  observable  - hence  sensing. 

Another  methodology  includes  deductive  reasoning 
which  is  the  case  of  looking  at  the  world  and  determining 
why  such  things  exist.  One  example  is  that  of  the 
movement  of  systems.  We  understand  that  objects  fall  to 
the  earth  and  deduce  that  something,  gravity,  causes  the 
objects  to  fall.  Many  of  the  deductions  on  object  motions 
come  from  the  mathematics  deduced  from  the  world,  such 
as  inertia. 

Once  we  have  defined  the  ontology,  we  now  can 
formulate  the  epistemology  of  information. 

4,2  Epistemology 

It  is  important  to  discuss  the  epistemology  of  a 
philosophical  argument  as  defined  through  the  ages  in 
positing  a theory.  The  epistemology  for  fusion  is  based  on 
the  ontology  chosen  which  includes  (1)  active  reasoning, 
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and  (2)  display  issues  such  as  semantics,  efficacy,  and  a 
taxonomy. 

4.2.1  Active  Reasoning 

In  the  case  presented  for  a user  to  interact  with  the 
fusion  system,  the  user  can  also  take  a proactive  role  in  the 
assessment  of  data  collection  (i.e.  level  4 - sensor 
management).  A user  can  act  in  a variety  of  abilities  such 
as  monitoring  a situation  in  an  active  or  passive  role  or 
planning  by  either  reacting  to  new  data  or  proactive 
control  over  the  fusion  system.  Thus,  the  human  has  to 
decide  in  which  way  to  convey  information:  such  as 
Monitor  (active  versus  passive)  or  Planner  (reactive  vs. 
proactive) 

When  a human  interacts  with  a system,  it  is 
important  the  ability  of  reasoning  be  available  for  the 
fusion  system.  The  human  has  the  abilities  to  quickly 
reduce  the  search  space  of  the  fusion  system  and  hence, 
guide  the  fusion  system  process.  Such  an  example  is 
when  the  human  cues  a fusion  system  to  look  for  an 
object  in  a certain  area  of  the  earth.  The  human  can 
provide  many  roles  to  interact  with  a fusion  system,  such 
as  predicting  target  motions,  reacting  to  new  information, 
and  proactively  cueing  the  system  to  search  for  things  that 
are  assumed  to  exist.  If  human  does  not  take  an  active 
role  in  monitoring  a fusion  system,  then  the  person  is 
acting  in  a passive  role,  shown  in  Figure  3. 

Predictive  - projective,  analytical  (anticipatory) 

Reactive  - immediate,  unthinking 

Proactive  - practical  / active  / thinking 

Passive  - waiting  / non-thought  (not  necessarily  active) 
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Figure  3.  Analysis  of  types  of  Human  Refinement. 


The  taxonomy  of  the  user  roles,  is  based  actions 
provided  to  the  user.  The  actions  are  a result  of  the  fusion 
system  interface  and  should  be  designed  into  future  fusion 
systems  so  as  to  accomplish  the  ontological  goals  of  the 
system. 

While  the  paper  addresses  many  ontological 
questions  concerning  the  human  interaction  with  a fusion 
system,  there  are  some  other  issues  associated  with  the 
inferencing  issues:  such  as  semantics,  taxonomy,  and 
efficacy  of  results. 

4.2.2  Display  / Interface 

The  display  interface  is  key  to  allowing  the  user  to  have 
control  over  the  data  collection  and  fusion  processing. 


[18]  Without  designing  a display  that  matches  the 
cognitive  perception  of  the  information,  it  is  difficult  for 
the  user  to  reason  over  the  fused  result.  While  many 
papers  and  books  address  the  interface  issues  (i.e.  see 
multimodal  interfaces),  it  is  of  concern  for  the  fusion 
community  to  address  the  cognitive  user  issues  to  ensure 
that  the  fusion  system  designed  is  to  emulate  the 
functional  roles  required  of  the  fusion  system. 

4.2.2.1  Semantics 

Developing  a framework  of  a user  refinement  system  must 
have  some  sort  of  semantics  or  interface  actions  that  allow 
the  system  to  coordinate  with  the  user.  Such  an  example 
is  a query  system  in  which  the  human  seeks  questions  and 
the  system  translates  these  requests  into  actionable  items. 

4.2.2.2  Efficacy 

Efficacy  is  an  important  concept  for  the  ontological 
questions  involving  a human  and  a computer  system.  If 
the  user  has  confidence  and  reliability  in  the  fusion 
system,  the  fusion  system  will  be  further  utilized  for  it  s 
capabilities.  Without  having  trust  in  automation,  the 
effects  of  the  inferencing  over  the  data  will  not  result. 

4.2.2.3  Taxonomy 

A taxonomy  [19]  is  a classification  of  something  - such  as 
algorithms,  processes,  and  things.  In  order  to  execute  the 
prioritization  information,  there  has  to  be  some  sort  of 
classification.  The  difficulty  is  that  each  user,  unless 
trained,  sees  the  world  slightly  differently  and  thus,  has  a 
different  classification  for  objects  and  processes.  In  order 
to  develop  efficient  and  effective  fusion  systems,  it  is 
important  to  develop  a taxonomy  that  relates  to  the 
ontological  perspectives  of  what  the  human  expects  in  the 
world.  Such  an  example  is  tracking  a target.  The 
expected  processes  are  detection,  recognition, 
classification,  and  identification.  If  the  object  is 
identified,  then  it  is  assumed  that  the  target  is  detected  and 
not  the  other  way  around.  Seeking  a valid  taxonomy  of 
objects  and  fusion  strategies  will  help  the  user  in 
determining  what  role  they  should  play  in  the  active 
reasoning  over  data  and  fused  information. 

5 Conclusions 

As  presented  in  this  article,  data  fusion  involves  the 
integration  and  application  of  many  types  of  data  and 
processing  techniques  to  satisfy  a user  need.  The  paper 
described  various  taxonomies  of  terminology  to  further 
the  ontological  questions.  The  ontological  questions 
surrounding  a fusion  system’s  ability  to  provide 
knowledge,  requires  a consistent  ontology,  epistemology, 
and  pedigree  information.  As  a summary,  Figure  4 shows 
the  relationship  between  the  ontological  issues  of 
deductive  logic  about  the  knowledge  in  the  world  and  the 
inductive  logic,  from  fusion  of  observations  about  the 
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Figure  4.  Ontological  Explanation  of  human  refinement  of  information  fusion.[20] 


world.  It  is  in  the  mind  of  the  user  in  which  an 
intersection  occurs.  The  data  observed  and  the  models 
processed  need  to  be  of  value  to  the  user  for  effective 
fusion  system  designs.  Future  research  will  involve 
extending  the  taxonomy  and  ontology  to  situational 
awareness[21]  to  integrate  the  human  and  fusion  system 
attributes. 
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